Project 2¶
2: Get the Data¶
Task #2a: Load the Telcom Customer Churn dataset into a DataFrame.
Task #2b: Confirm the import by executing the head method on your DataFrame.
Task #2c: Determine the ‘shape’ of your DataFrame. Then list the fields with their datatypes using the info method.
# 2a
import pandas as pd
churn = pd.read_csv('churn.csv', delimiter = ',', thousands = ',')
# 2b
churn.head()
# 2c
print(churn.shape)
churn.info() #checking number of null values
churn.dropna(inplace = True) # eliminate null values
churn.info() # same number of rows as previous, showing that no null values were dropped... interesting
3: Clean the Data¶
Task #3a: For customers with a missing Churn status (neither Yes, or No), fill-in the values to be ‘No’.
#3a
churn.Churn.value_counts() #checking again for null values in Series...5174 + 1869 = 7038 which shows that there are values not being read
churn.Churn = churn.Churn.apply(lambda x: 'Yes' if x == 'Yes' else 'No' ) # recoding
churn.Churn.value_counts() #still no change
Task #3b: Drop any rows with more than one missing value.
#3b Dropped any N/A's
churn.dropna(inplace = True)
churn.customerID.value_counts() #still no change
Task #3c: Eliminate any duplicate records.
#3c no duplicate customer ID's but for good measure, showing any duplicates:
churn.customerID[churn.customerID.duplicated()]
# row count matches 7043
churn.columns
churn.InternetService.value_counts() # totals 7043
churn.OnlineBackup.value_counts()
churn.OnlineSecurity.value_counts()
Task #3d: Create a new column within the DataFrame called ‘Techsavvy’. Rows within this column will have a value of either high, medium, or low. High Techsavvy customers use both online security and online backup and have a Fiber Optic internet service. Medium Techsavvy customers also use Fiber Optic internet service and online security, but not online backup. All other customers are low Techsavvy. As we cannot really tell the level of tech-savvyness for customers who do not use the internet service (i.e., are ‘no’), do not give them any ‘Techsavvy’ rating – instead, put a placeholder value of ‘unknown’ in the Techsavvy column. Note: exclude customers with an ‘unknown’ Techsavvy value from any visualizations (see below).
# 3d creating new column
def tech_savy(x):
if x.InternetService == 'Fiber optic':
if x.OnlineSecurity =='Yes' and x.OnlineBackup == 'Yes':
return 'High'
elif x.OnlineSecurity =='Yes' and x.OnlineBackup == 'No':
return 'Medium'
else:
return 'Low'
elif x.InternetService =='DSL':
return 'Low'
elif x.InternetService =='No':
return 'Unknown'
churn['Techsavvy'] = churn.apply(tech_savy, axis = 1)
churn.Techsavvy.value_counts() #7043 rows...all rows accounted for
Task #3e: Create a new column within the DataFrame called ‘Streamer’. Rows within this column will have a value of ‘streamer’ if they subscribe to both StreamingTV and StreamingMovies. Otherwise, a customer will have a streamer value of non-streamer. Again, preclude any customers that do not subscribe to any internet service – give these customers the value of ‘unknown’. Note: exclude customers with an ‘unknown’ Streamer’ value from any visualizations (see below).
# 3e creating a new column
def streamer(x):
if x.InternetService == 'No':
return 'Unknown'
else:
if x.StreamingMovies == 'Yes' and x.StreamingTV == 'Yes':
return 'Streamer'
else:
return 'Non-streamer'
churn['Streamer'] = churn.apply(streamer, axis = 1)
churn.Streamer.value_counts()
Task #3f: Replace all occurrences of ‘No internet service’ with ‘NA’.
#3f replacing all No in internet service with NA
churn.InternetService = churn.InternetService.map({'Fiber optic': 'Fiber optic', 'DSL': 'DSL', 'No': 'NA'})
churn.InternetService.value_counts()
Task #3g: Recode ‘Bank transfer (automatic)’ and ‘Credit card (automatic)’ to be ‘Automatic’
churn.PaymentMethod.value_counts() # 3g checking for all values
# maping automatic payments to Automatic
churn.PaymentMethod = churn.PaymentMethod.map({'Electronic check': 'Electronic check', 'Mailed check': 'Mailed check', 'Bank transfer (automatic)' : 'Automatic', 'Credit card (automatic)': 'Automatic'})
churn.PaymentMethod.value_counts() # Automatic does equal total of both previous credit card and bank transfer
Task #3h: Create an explicit row index for your DataFrame using the customerID field. Name this index CustomerID.
#3h setting index
churn.set_index('customerID', inplace = True)
churn.head()
Task #3i: Identify any potential outliers in the column MonthlyCharges. Plot the values to help identify any outliers. If you find potential outliers, change the monthly charge to match the ceiling amount.
churn.MonthlyCharges #3i
#3i no outliers identified ??
import seaborn as sns
import numpy as np
import plotly.express as px
sns.distplot(churn.MonthlyCharges)
fig = px.histogram(churn, x = churn.MonthlyCharges, nbins = 50)
fig.show()
Task #3j: Perform a cross-check of tenure multiplied by MonthlyCharges is equal TotalCharges. If the cross-check fails to be within a 10\% range of the recorded TotalCharges, then drop the entire record.
# converting all columns to number data types
#3j
churn.tenure = churn.tenure.astype(int)
churn.MonthlyCharges = churn.MonthlyCharges.astype(float)
# error at position 488 ---- BAD DATA ---drop rows with " " for TotalCharges
churn.TotalCharges = pd.to_numeric(churn.TotalCharges)
#identifying number of rows with blank TotalCharges
churn.TotalCharges[churn.TotalCharges != " "] # row count 7032 which means there are 11 missing values
churn.TotalCharges.dropna(inplace = True) #ineffective
# row number matches
churn.shape # showing dropna is not working
#changed all " " to 0 in order to transform column
churn.TotalCharges[churn.TotalCharges == " "] = 0
churn.TotalCharges[churn.TotalCharges != " "] # row count changed to 7043 showing that the 11 missing values have been recoded to 0
churn.TotalCharges = pd.to_numeric(churn.TotalCharges)
# creating a new column cross-check, this column is creating with a function that assigns Total Charges. If Total Charges = 0 then Cross Check is NaN, if Total Charges does not fall with 10% of monthly charges x tenure then Cross Check is assigned NaN.
# Cross Check rows assigned NaN will be removed later
def Cross_Check(x):
if x.TotalCharges == 0:
return 'NaN'
else:
a = (x.tenure*x.MonthlyCharges)/x.TotalCharges
if a > .9 and a < 1.10:
return a
else:
return 'NaN'
churn['cross_check'] = churn.apply(Cross_Check, axis = 1)
churn.cross_check.value_counts() #7042 total, 614 :1, 397: NaN
#drop all rows not within 10% range
churn.drop(churn[churn['cross_check'] == 'NaN'].index, inplace = True)
# rows check out 7043 - 397 (NaN) = 6646
churn.info()
4: Structure¶
Task #4a: Categorize the ‘tenure’ field across four intervals. Ensure the interval cutoffs provide a balanced number (or close to that) of customers within each category and choose appropriate brief labels for each category. (Hint: you can use cut/qcut). Generate a visualization to display the distribution of customers across the categories to confirm the categorization has been performed correctly. Be careful to preserve the ‘tenure’ field, as it will be needed for our Logistic Regression analysis. Place the categorized tenure field (name this field: ‘TenureCat’) immediately to the right of the tenure field within the DataFrame.
#4a creating qcut of tenure into equalish sized bins
tenure = ['Infant','Child', 'Teen', 'Adult']
churn['TenureCat'] = pd.qcut(churn.tenure, 4, labels=tenure)
Note: I have create 4 Tenure Categories named: Infant, Child, Teen, Adult¶
churn.TenureCat.value_counts()# total of rows = 6646
# reordering columns
churn.columns
churn = churn[['gender', 'SeniorCitizen', 'Partner', 'Dependents', 'tenure','TenureCat',
'PhoneService', 'MultipleLines', 'InternetService', 'OnlineSecurity',
'OnlineBackup', 'DeviceProtection', 'TechSupport', 'StreamingTV',
'StreamingMovies', 'Contract', 'PaperlessBilling', 'PaymentMethod',
'MonthlyCharges', 'TotalCharges', 'Churn', 'Techsavvy', 'Streamer',
'cross_check']]
churn.columns
churn.head()
import plotly.express as px
fig = px.histogram(churn, x = churn.TenureCat, color = churn.TenureCat)
fig.show()
Task #4b: For each of the tenure categories, provide a count of customers across the different contract types (one year, two year, etc.). Also report the average monthly charge across the different contract types. Plot this data in an appropriate visualization to promote visual comparison across the tenure categories.
#4b for each TenureCat provide a count of customers across the different contract types
px.histogram(churn, x = churn.TenureCat, color = churn.Contract, barmode = 'group')
#4b Continued: showing average MonthlyCharges by Contract and TenureCat
u = churn.groupby(['Contract', 'TenureCat'])['MonthlyCharges'].mean()
u = u.unstack()
u
#graph to compare monthly charges across TenureCat by contract type
import plotly.graph_objects as go
import plotly.figure_factory as ff
table_data = u
fig = ff.create_table(table_data, height_constant=50)
Contract_type = u.columns
U1 = u.iloc[0]
U2 = u.iloc[1]
U3 = u.iloc[2]
trace1 = go.Bar(x=Contract_type, y=U1, xaxis='x2', yaxis='y2',
marker=dict(color='lightblue'),
name='Month<br>to<br>Mont')
trace2 = go.Bar(x=Contract_type, y=U2, xaxis='x2', yaxis='y2',
marker=dict(color='salmon'),
name='One<br>Year')
trace3 = go.Bar(x=Contract_type, y=U3, xaxis='x2', yaxis='y2',
marker=dict(color='indianred'),
name='Two<br>Year')
fig.add_traces([trace1, trace2, trace3])
fig['layout']['xaxis2'] = {}
fig['layout']['yaxis2'] = {}
fig.layout.yaxis.update({'domain': [0, .20]})
fig.layout.yaxis2.update({'domain': [.40, 1]})
fig.layout.yaxis2.update({'anchor': 'x2'})
fig.layout.xaxis2.update({'anchor': 'y2'})
fig.layout.yaxis2.update({'title': 'Monthly Charge in Dollars'})
fig.layout.margin.update({'t':75, 'l':50})
fig.layout.update({'title': 'Breakdown of Tenure Monthly Charge'})
fig.layout.update({'height':800})
fig.show()
#7a showing the count of each churn status
px.histogram(churn, x = churn.Churn, color = churn.Churn)
Task #7b: Generate a barplot comparing MonthlyCharges against InternetService.
#7b Generate a barplot comparing MonthlyCharges against InternetService
from plotly.subplots import make_subplots
# groupby to get the appropriate data
sum_IS = churn.groupby('InternetService')['MonthlyCharges'].sum()
mean_IS = churn.groupby('InternetService')['MonthlyCharges'].mean()
fig = make_subplots(rows=1, cols=2)
fig.add_bar(x = sum_IS.index, y=sum_IS,
marker=dict(color="MediumPurple"), text = sum_IS, textposition ='outside',
name="Total Monthly Charges", row=1, col=1)
fig.add_bar(x = sum_IS.index, y=mean_IS,
marker=dict(color="LightSeaGreen"),
name="Average Monthly Charges",text = sum_IS, textposition = 'outside', row=1, col=2, )
fig.layout.title.text = "Monthly Charges by Internet Service"
fig.show()
Task #7c: Generate a violinplot comparing Churn status against InternetService.
#7c Generate a violinplot comparing MonthlyCharges against InternetService. Select Churn as the hue.
mean_IS = churn.groupby(['InternetService', 'Churn'])['MonthlyCharges'].mean()
mean_IS = mean_IS.unstack(1)
y_1 = mean_IS.Yes
y_2 = mean_IS.No
fig = go.Figure()
fig.add_trace(go.Violin(
y=y_1,
name='Churn = Yes',
box_visible=True,
meanline_visible=True))
fig.add_trace(go.Violin(
y=y_2,
name='Churn = No',
box_visible=True,
meanline_visible=True))
fig.layout.title.text = "Monthly Charges by Churn Status and InternetService"
fig.show()
mean_IS
Task #7d: Generate a series of plots comparing churn status against tenure, MonthlyCharges, TotalCharges, Techsavvy, SeniorCitizen, TenureCat, and Streamer.
Based on the visualizations performed so far, provide at least 3 preliminary observations – for example, do these visualizations provide any indicators of possible good predictors of Churn status? Please record your observations within the Notebook.
#7d Generate a series of plots (Churn vs TotalCharges, Techsavvy, MonthlyCharges, SeniorCitizen, TenureCat, Streamer)
import plotly.graph_objects as go
ch_totalcharge = churn.groupby('Churn')['TotalCharges'].mean()
px.bar(x = ch_totalcharge.index, y = ch_totalcharge)
x = ch_totalcharge.index
y = [ch_totalcharge[0],ch_totalcharge[1]]
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon'],
textposition='auto',
)])
fig.update_layout(title_text='Churn Status vs Average Total Charge')
fig.show()
ch_techsavvy = churn.groupby(['Techsavvy','Churn'])['Churn'].count()
ch_techsavvy = ch_techsavvy.unstack(1)
x = ch_techsavvy.index
y_1 = ch_techsavvy.No
y_2 = ch_techsavvy.Yes
fig = go.Figure()
fig.add_trace(go.Bar(
x=x, y=y_1,
text=y_1,
name = 'No Churn',
marker_color = 'lightblue',
textposition='auto',
))
fig.add_trace(go.Bar(
x=x, y=y_2,
text=y_2,
name = 'Yes Churn',
marker_color = 'salmon',
textposition='auto',
))
fig.update_layout(title_text='Churn Status vs TechSavvy Status')
fig.show()
ch_monthlycharges = churn.groupby('Churn')['MonthlyCharges'].sum()
x = ch_monthlycharges.index
y = [ch_monthlycharges[0],ch_monthlycharges[1]]
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon'],
textposition='auto',
)])
fig.update_layout(title_text='Churn Status vs Total Monthly Charges')
fig.show()
## Clo suggestion: easier to see that there is no pattern I think, need to change the label if we go with that
churn['MonthlyCat']=pd.qcut(churn['MonthlyCharges'], q=20, labels=['1', '2', '3', '4', '5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20'])
px.histogram(churn, x = churn.MonthlyCat, color = churn.Churn, barmode = 'group')
churn.drop('MonthlyCat', axis = 1, inplace = True)
churn.SeniorCitizen = churn.SeniorCitizen.apply(lambda x: 'No' if x == 0 else 'Yes')
ch_seniorcitizen = churn.groupby('SeniorCitizen')['Churn'].count()
x = ch_seniorcitizen.index
y = [ch_seniorcitizen[0],ch_seniorcitizen[1]]
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon'],
textposition='auto',
)])
fig.update_layout(title_text='Churn Status vs Senior Citizen')
fig.show()
ch_tenurecat = churn.groupby(['TenureCat', 'Churn'])['Churn'].count()
ch_tenurecat = ch_tenurecat.unstack(1)
x = ch_tenurecat.index
y_1 = ch_tenurecat.No
y_2 = ch_tenurecat.Yes
fig = go.Figure()
fig.add_trace(go.Bar(
x=x, y=y_1,
text=y_1,
name = 'No Churn',
marker_color = 'lightblue',
textposition='auto',
))
fig.add_trace(go.Bar(
x=x, y=y_2,
text=y_2,
name = 'Yes Churn',
marker_color = 'salmon',
textposition='auto',
))
fig.update_layout(title_text='Churn Status vs TenureCat Status')
fig.show()
ch_streamer = churn.groupby(['Streamer', 'Churn'])['Churn'].count()
ch_streamer = ch_streamer.unstack(1)
x = ch_streamer.index
y_1 = ch_streamer.No
y_2 = ch_streamer.Yes
fig = go.Figure()
fig.add_trace(go.Bar(
x=x, y=y_1,
text=y_1,
name = 'No Churn',
marker_color = 'lightblue',
textposition='auto',
))
fig.add_trace(go.Bar(
x=x, y=y_2,
text=y_2,
name = 'Yes Churn',
marker_color = 'salmon',
textposition='auto',
))
fig.update_layout(title_text='Churn Status vs Streamer Status')
fig.show()
- Fiber Optic Internet service accounts for almost half of all revenues according to graph in 7b
- Customers who have NOT churned are spending more than those who have abandoned service
- Customers that are 'Medium' Techsavvy have a much higher rate of churn (102/237 = 43%)
- TechSavvy 'High' customers have a very low churn rate
- Newer customers (Infants) have a much higher churn rate, most likely due to the high percentage of month-to-month contracts
- Senior Citizens seems also a good predictor for churn status
Task #7e: Calculate categorical means (i.e., means for each category) for MonthlyCharges across the InternetService, TenureCat’ and Techsavvy categories. Also, calculate categorical means for Tenure across the InternetService, Techsavvy and SeniorCitizen categories. Hint: use groupby. Plot the two most interesting results.
#7e categorical means MonthlyCharges: InternetService, TenureCat, TechSavvy. Tenure: InternetService, TenureCat, SeniorCitizen
mc_internetservice = churn.groupby('InternetService')['MonthlyCharges'].mean()
x = mc_internetservice.index
y = mc_internetservice
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon', 'indianred', '#330C73'],
textposition='auto',
)])
fig.update_layout(title_text='MonthlyCharges vs InternetService')
fig.show()
churn.InternetService
mc_tenurecat = churn.groupby('TenureCat')['MonthlyCharges'].mean()
x = mc_tenurecat.index
y = mc_tenurecat
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon', 'indianred', '#330C73'],
textposition='auto',
)])
fig.update_layout(title_text='MonthlyCharges vs Tenure Category')
fig.show()
#thoughts: is there any way to put medium between low and high? to change the order
mc_techsavvy = churn.groupby('Techsavvy')['MonthlyCharges'].mean()
x = mc_techsavvy.index
y = mc_techsavvy
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon', 'indianred', '#330C73'],
textposition='auto',
)])
fig.update_layout(title_text='MonthlyCharges vs Techsavvy')
fig.update_layout(xaxis={'categoryorder':'mean descending'})
fig.show()
tn_internetservice = churn.groupby('InternetService')['tenure'].mean()
x = tn_internetservice.index
y = tn_internetservice
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon', 'indianred'],
textposition='auto',
)])
fig.update_layout(title_text='Average Tenure vs InternetService')
fig.show()
tn_tenureCat = churn.groupby('TenureCat')['tenure'].mean()
x = tn_tenureCat.index
y = tn_tenureCat
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon', 'indianred', '#330C73'],
textposition='auto',
)])
fig.update_layout(title_text='Tenure vs Tenure Category')
fig.show()
tn_seniorcitizen = churn.groupby('SeniorCitizen')['tenure'].mean()
x = tn_seniorcitizen.index
y = tn_seniorcitizen
fig = go.Figure(data=[go.Bar(
x=x, y=y,
text=y,
marker_color = ['lightblue', 'salmon'],
textposition='auto',
)])
fig.update_layout(title_text='Senior Citizen vs Tenure Category')
fig.show()
- The plot where we can see the correlation between the Monthly charges and the Tech savy is interesting because we can see that the monthly charges is bigger when the customer is more tech savvy.
Task #7f: Generate a single kdeplot plot comparing the distributions of MonthlyCharges and (TotalCharges divided by tenure).
#7f Generate a single kdeplot plot comparing the distributions of MonthlyCharges and (TotalCharges divided by tenure)
import plotly.figure_factory as ff
a = churn.MonthlyCharges
b = churn.TotalCharges/churn.tenure
colorscale = ['#7A4579', '#D56073', 'rgb(236,158,105)', (1, 1, 0.2), (0.98,0.98,0.98)]
fig = ff.create_2d_density(
a, b, colorscale=colorscale,
hist_color='rgb(255, 237, 222)', point_size=3
)
fig.layout.update({'height':800})
fig.layout.update({'width':1000})
fig.update_layout(title_text='Monthly Charges vs TotalCharges/Tenure')
fig.show()
#
import matplotlib.pyplot as plt
plt.rc("font", size=14)
import seaborn as sns
plt.figure(figsize=(15,8))
ax = sns.kdeplot(a, color="lightblue", shade=True)
sns.kdeplot(b, color="salmon", shade=True)
plt.legend(['Monthly Charges', 'Total Charges over Tenure'])
plt.title('Density Plot')
ax.set(xlabel='Monthly_Charges')
plt.xlim(-10,120)
plt.show()
Task #7g: Generate a kdeplot comparing tenure and monthly charges. As these variables are clearly on different scales, you will need to first normalize the values in both variables (use the min-max normalization approach) and also remove any distorting outliers.
#7g Density plot comparing MonthlyCharges vs Tenure
import sklearn.preprocessing
a = churn.MonthlyCharges
b = churn.tenure
x = sklearn.preprocessing.minmax_scale(a, feature_range=(0, 1), axis=0, copy=True)
y = sklearn.preprocessing.minmax_scale(b, feature_range=(0, 1), axis=0, copy=True)
colorscale = ['#7A4579', '#D56073', 'rgb(236,158,105)', (1, 1, 0.2), (0.98,0.98,0.98)]
fig = ff.create_2d_density(
x, y, colorscale=colorscale,
hist_color='rgb(255, 237, 222)', point_size=3
)
fig.layout.update({'height':800})
fig.layout.update({'width':1000})
fig.update_layout(title_text='Monthly Charges vs Tenure')
fig.show()
import matplotlib.pyplot as plt
plt.rc("font", size=14)
import seaborn as sns
plt.figure(figsize=(15,8))
ax = sns.kdeplot(x, color="lightblue", shade=True)
sns.kdeplot(y, color="salmon", shade=True)
plt.legend(['Tenure', 'Monthly Charges'])
plt.title('Density Plot')
ax.set(xlabel='Monthly_Charges')
plt.xlim(-.25,1.25)
plt.show()
Task #7h: Generate a plot investigating whether the dataset has a gender bias.
#7h countplot of male vs female
import plotly.express as px
px.histogram(churn, x = churn.gender, color = churn.gender)
- There is no gender bias for dataset
Task #7i: Generate a series of box plots showing MonthlyCharges against these fields: Techsavvy, Streamer, SeniorCitizen, and InternetService.
px.box(churn, x='Techsavvy', y='MonthlyCharges', color = 'Techsavvy')
px.box(churn, x='Streamer', y='MonthlyCharges', color = 'Streamer')
px.box(churn, x='SeniorCitizen', y='MonthlyCharges', color='SeniorCitizen')
px.box(churn, x='InternetService', y='MonthlyCharges', color='InternetService')
8: Feature Selection¶
For our intended analytical method – binary linear regression – the independent variables should be independent of each other: that is, the model should have little or no multicollinearity. We would therefore exclude the following fields from further analysis: tenure, InternetService, OnlineSecurity, OnlineBackup, DeviceProtection, TechSupport, StreamingTV, and StreamingMovies, as we have created other fields based on these fields. Moreover, we would expect that the TotalCharges would be highly correlated with tenure and MonthlyCharges – accordingly, we would also exclude TotalCharges from our model.
For brevity of analysis, we will now move forward to our modeling stage with a focus on only the following fields:
- Dependent variable: Churn
- Independent variables: SeniorCitizen, Streamer, Techsavvy, TenureCat, MonthlyCharges, PaymentMethod. Go ahead and drop all other fields from the DataFrame (but keep the row index).
# 8 Creating new dataframe
drop_list = ['tenure', 'InternetService', 'OnlineSecurity', 'OnlineBackup', 'DeviceProtection', 'TechSupport', 'StreamingTV', 'StreamingMovies', 'TotalCharges', 'Partner','Dependents','PhoneService','MultipleLines','PaperlessBilling', 'cross_check', 'Contract','gender']
churn_new = churn.drop(drop_list, axis = 1)
churn_new.to_csv('churn_new')
9: Modeling¶
Task #9a: For logistic regression, categorical variables need to be dummied (create dummy variables). As explanation, you cannot directly feed into the model labelled data (e.g., low, medium, high), as it has no clue what you are talking about (i.e., the order) and has trouble comparing values across variables. To address this issue, we need to recode the categorical variables. Pandas has a specific capability to help with this type of recoding – the getdummies method. Application of this method will produce an Indicator matrix (DataFrame), which will need to be merged / concatenated back with the main DataFrame. Also, as several categorical fields feature more than two categories, you will need to create special fields within the DataFrame to accommodate this.
Task #9b: Check to ensure all categorical columns have 1,0s (True/False) values. You can do this through visual inspection, using head.
#9a and 9b
churn_dummies = pd.get_dummies(churn_new)
churn_dummies = churn_dummies.drop(['Churn_No', 'SeniorCitizen_No'], axis = 1)
churn_dummies.head()
churn_dummies.loc[:, churn_dummies.columns != 'MonthlyCharges'].astype('category')
Task #9c: With the dataset now fully prepped for modeling, perform a binary linear regression analysis. Use the scikit-learn package: specifically, the sklearn.linear_model.LogisticRegression class. Given the technical nature of the official documentation, you may find a blog post demonstrating application of this method of some assistance (for example, see below). Note: as the focus of this project is not on mastering methods, it is not necessary to have the application of this method reflect all best practices aimed at model evaluation (e.g., recursive feature elimination, analysis of ROC, and confusion matrix analysis).
https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8
https://scikit-learn.org/stable/index.html
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
#9c
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import RFE
from imblearn.over_sampling import SMOTE
import numpy as np
X = churn_dummies.loc[:, churn_dummies.columns != 'Churn_Yes']
y = churn_dummies.loc[:, churn_dummies.columns == 'Churn_Yes']
#checking variance_inflation_factors
from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
vif["features"] = X.columns
#checking variance_inflation_factors
vif
X_2 = X.drop(['Techsavvy_Medium', 'Streamer_Unknown', 'TenureCat_Adult', 'PaymentMethod_Mailed check' ], axis = 1)
import statsmodels.api as sm
logit_model=sm.Logit(y,X_2)
result=logit_model.fit()
print(result.summary())
10¶
vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X_2.values, i) for i in range(X_2.shape[1])]
vif["features"] = X_2.columns
vif
#droping Streamer_Non-streamer for multicollinearity
#droping PaymentMethod_Automatic because of P-value > .05
X_3 = X_2.drop(['Streamer_Non-streamer', 'PaymentMethod_Automatic' ], axis = 1)
X_3
logit_model=sm.Logit(y,X_3)
result=logit_model.fit()
print(result.summary())
- Newer customers (infants and children) are more liable to churn
- This information verifies 2 previous Exploratory Data Analysis charts: TenureCat vs Churn, the density plots of Tenure vs Monthly Charge